Purpose / Description

Brings the existing observal migrate CLI capabilities into the settings, so a super_admin can move an Observal instance's data in a few clicks without using the CLI.

Today, migrating data requires an operator to know and correctly sequence CLI commands, manage connection strings, and shuttle archive files between machines by hand. This feature reduces that to three operations driven from the UI:

1. Export — the server dumps its own connected PostgreSQL data, ClickHouse telemetry data, or both into downloadable .tar.gz archives.
2. Import — the operator uploads previously-exported archives and the server loads them into its own databases with automatic org ID rewriting.
3. Validate — standalone checksum, row-count, and FK reference verification on archives without importing.

All operations run as background jobs on the existing arq worker, with progress streamed to the UI and artifacts offered for download when ready. The existing CLI commands remain fully functional — both the UI and CLI share the same migration service core.

Approach

Architecture

• Shared Migration Service (observal-server/services/migration/): Extracted all migration logic from cmd_migrate.py into a shared, import-light package with no typer/rich/FastAPI dependencies at module scope. Both the CLI and the server import from this core.

• REST API (api/routes/admin/migrate.py): 8 endpoints for export/import/validate (start + status polling + artifact download + current-org). All require super_admin role. Concurrency check prevents duplicate jobs. Upload validation checks magic bytes and size.

• Background Job (jobs/migration.py): run_migration_job loads the job row, creates artifact directory (0o700), dispatches to the shared service, writes progress to DB (throttled 1s), and handles success/failure with audit events. purge_migration_artifacts cron deletes expired artifacts.

• Frontend (web/src/pages/admin/settings.tsx, web/src/pages/admin/dashboard/components/migrate-*.tsx): Data Migration section in Settings (super_admin-gated, between Registered Agents Only and Data Retention), tabbed dialog (Export/Import/Validate), spinner-based progress, result display with download buttons.

• PostgreSQL Model + Alembic Migration: MigrationJob table with enums for operation/scope/status, progress fields, result JSON, and artifact metadata.

Key Design Decisions

• CLI imports server's shared core (Option A from design doc): True single source of truth. CLI keeps operating on operator-supplied URLs while server uses its own connections.
• Polling over WebSockets: TanStack Query refetchInterval (1.5s) while job is active. Simpler, survives reconnects, matches existing InsightReport pattern.
• Auto org rewrite on import: Target instance's org is auto-detected and all org references are rewritten, so imported data is immediately visible.
• Telemetry packed into single tar.gz: Export produces exactly two files (pg_export.tar.gz + telemetry_export.tar.gz) for clean UX.

How Has This Been Tested?

1. Unit tests (tests/test_migration_*.py, observal_cli/tests/test_cmd_migrate.py):

   • Service import isolation (no framework deps)
   • CLI commands invoke shared service correctly (11 tests)
   • REST API endpoints (202, 409, 422, 403, audit events)
   • Background job lifecycle (status transitions, throttling, errors)
   • Artifact security (token minting, expiry, path traversal protection)

2. Property-based tests (Hypothesis, tests/test_migration_properties.py):

   • 15 properties covering: round-trip preservation, FK-safe ordering, schema tolerance, idempotent CH import, org rewrite, row accounting, checksum failure, validation, state machine, credential exclusion, token expiry, TTL purge, role denial

3. Integration tests (tests/test_migration_integration.py):

   • Export→validate round-trip
   • Idempotent re-import (ON CONFLICT DO NOTHING)
   • TTL purge removes aged directories

4. Manual E2E testing against local Docker stack:

   • Export (PostgreSQL only, PostgreSQL+ClickHouse both) ✓
   • Import with org rewrite ✓
   • Validate ✓
   • Download artifacts ✓
   • Concurrency check (409) ✓

Checklist

• You have a descriptive commit message with a short title (first line, max 50 chars).
• You have commented your code, particularly in hard-to-understand areas
• You have performed a self-review of your own code
• UI changes: include screenshots of all affected screens (in particular showing any new or changed strings)

Licenses

AI Assistance

• Yes: Kiro (Claude)
• Was the generated code manually reviewed and tested?